THE SYMPLECTIC REPRESENTATION OF THE 
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MUSTAFA KORKMAZ 

Abstract. Any nontrivial homomorphism from the mapping class group 
of an orientable surface of genus g > 3 to GL(2(;, C) is conjugate to the 
standard symplectic representation. It is also shown that the mapping 
class group has no faithful linear representation in dimensions less than 
or equal to 3g — 3. 



1. Introduction and the results 

For a compact connected orientable surface S of genus g with a finite (pos- 
sibly empty) set of marked points in the interior, we define the mapping class 
group Mod(S') to be the group of isotopy classes of orientation-preserving 
self-diffeomorphisms of S. Diffeomorphisms and isotopies are assumed to 
be the identity on the boundary and on the marked points. Gluing a disk 
along each boundary component of S and forgetting the marked points give 
a closed surface S and a natural surjective map Mod(5) — )■ Mod(5) between 
the mapping class groups. 

After fixing a basis for the first (integral) homology Hi{S,'L) of «S, the 
action of Mod(5) on Hi{S,'L) gives rise to a surjective homomorphism 
Mod(5) Sp(25,Z). Precomposing this map with Mod(S') Mod(5) 
and postcomposing with the inclusion Sp(2gf,Z) GL(2(7, C) give a map 
P : Mod(S') —7- GL(2(7, C). Our first result shows that the map P is the only 
nontrivial representation of Mod(S') in this dimension: 

Theorem 1. Let g > 3 and let <p '■ Mod(S') — )• GL(2(jr, C) be a group ho- 
momorphism. Then (p is either trivial or conjugate to the homovnorphism 
P : Mod(5) ^ GL(25,C). 

We say that two homomorphisms ■01 and 7/^2 from a group G to a group 
H are called conjugate if there exists an element h £ H such that ^'2(2^) = 
hTpi{x)h~^ for all x G G. 

One of the outstanding unsolved problems in the theory of mapping class 
groups is the existence of a faithful representation Mod(S') — t- GL(n,C). 
Our second result shows that in dimensions n < 3g — 3, there is no faithful 
linear representation of the mapping class group. 

Theorem 2. Let g > 3 and let n < 3g — 3. Then there is no injective 
homomorphism Mod(5) — )■ GL(n,C). 
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The linearity problem for the braid group, which is the mapping class 
group of a disk with marked points in the context of this paper, was solved 
by Bigelow [3] and also by Krammer [161 [T7] . In [13] , using the linearity of 
braid groups, the author proved that the mapping class group of the closed 
surface of genus 2 and of sphere with marked points are linear. These results 
were also obtained by Bigelow-Budney j4]. 

The first result on the nonexistence of faithful linear representations of the 
mapping class group was obtained by Farb-Lubotzky-Minsky [2] who proved 
that no homomorphism from a subgroup of finite index of the mapping class 
group into GL(n, C) is injective for n < — 1. This was improved first 
by Funar [8] who showed that every map from the mapping class group into 
SL(n, C) has finite image for n < \J g + 1. This was improved further by 
Franks-Handel in [7] and by the author in |12j . which can be rephrased as 
follows: 

Theorem 3. Let g > 2, n < 2g - 1, and let (/) : Mod(5) GL(n, C) 

be a homomorphism. Then (p factors through the natural quotient map 
Mod(S) — )• i?i(Mod(S'), Z). In particular, lm(0) is trivial if g > 3, and 
is a quotient of the cyclic group 7L\q of order 10 if g = 2. 

We state a few algebraic corollaries to above theorems, some of which 
might be known to the experts. 

Corollary 4. If g > 3 then up to the conjugation by an element ofGh{2g, C) 
there are exactly two homomorphisms Sp(2(jr, Z) — t- GL(2(7,C), the trivial 
homomorphism and the injection given by the inclusion. 

Corollary 5. Ifg > 3 andn < 2g then every homomorphism from PSp{2g,Z) 
to GL(n,C) is trivial. 

Corollary 6. If g > 3 and if Q is a finite quotient ofMod{S), then every 
homomorphism Q — )■ GL(2(7, C) is trivial. 

In order to prove these corollaries, consider the composition the given 
homomorphism H — >• GL(2(7, C) with the natural surjective map Mod(S') — )■ 
H and apply Theorem [1] or Theorem [3l 

Here is an outline of the paper. In Section [2l we give some lemmas on 
matrices. Section [3] gives the relevant background from the theory of map- 
ping class groups and surface topology. Section [J] investigates the properties 
of eigenvalues and eigenspaces of the image of a Dehn twist about a nonsep- 
arating simple closed curve. The main result in this section is Lemma |4.7[ 
which constitutes a major step in the proofs of both theorems. The first main 
result. Theorem [11 is proved in Section [5l We give the proof ior g > 4 first, 
and an then outline for g = 3. Finally, Theorem [2] is proved in Section [6l 
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In this section, we give the necessary definitions, setup the notation and 
prove some useful lemmas on matrices. We denote by S the closed surface 
obtained from S by gluing a disk along each boundary component. 

Throughout the paper, the letters U and U will always denote the 2x2 



Definition 2.1. For each i = 1,2, . . . , g, we define the matrices Ai to be the 
2g X 2g block diagonal matrix Diag(/2, . . . , I2,U, I2, ■ ■ ■ , I2) and Bi to be the 
2g X 2g block diagonal matrix Diag(/2, ■ ■ ■ , I2,U , I2, ■ ■ ■ , 12)- Here, U and U 
are in the i^^ block on the diagonal, and I2 is the 2x2 identity matrix. 

We note that Ai (resp. Bi) is the symplectic matrix of the action of the 
Dehn twist ta^ (resp. t;,.) on the first homology group of S with respect to the 
basis {ai, bi, a2,b2, . . . , ag,bg} Hi{S, Z), where ai and bi are the (oriented) 
simple closed curves given in Figure [2j Recall that when we consider a Dehn 
twist about a curve, the orientation of the curve is unimportant. 

Lemma 2.2. Let X, Y and Z be, respectively, 2 x k, k x 2 and 2x2, 
matrices with entries in C. 



Lemma 2.3. Let X, Y and Z be matrices with entries in C such that the 
given multiplication are defined: 

(i) // AiX = BiX = X for all i, then X = 0. 

(ii) // YAi = YBi = Y for all i, then Y = 0. 

(iii) If ZAi = AiZ and ZBi = BiZ for all i, then Z is equal to a 
diagonal matrix Diag(ai/2, 02-^2) • • • ) 0^3-^2) for some ai G C. 

Proof. This lemma is a slight generalization of Lemma 12.21 and may be 
proved easily by induction on g. □ 



2. Preliminaries in linear algebra 




(i) // UX = UX = X, then X = 0. 

(ii) // YU = YU = Y, then Y = 0. 

(iii) // ZU = UZ and ZU = UZ, then Z = al. 



Proof. The proof is straight forward. 



□ 



Lemma 2.4. Let X G GL(2,C). Suppose that XU = UX and XUX = 
UXU. Then X = U. 
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Proof. Let X = ^ ^ ^ / ' '^^^ equality X[/ = UX implies that c = 
and d = a, so that X = ^ ^ o ) " t^sX a / 0. Now the equality 

XUX = UXU gives the equations 

a(o — 6) = a — b, 

= 2a — b, 
b{2a -b) = b. 

The only solution of these equations is a = 6 = 1. □ 

Lemma 2.5. Let X G GL(25r, C). Suppose that 

(i) all eigenvalues of X are equal to 1, 

(ii) XAi = AiX for all i = 1,2, g, 

(iii) XBj = BjX for all j = 2,3, . . . , g, and 

(iv) XBiX = BiXBi. 

Then X = Ai. 

Proof. If g = 1 then the lemma reduces to Lemma 12.41 So suppose that 
5 > 2. 

/ Xi X2 \ 

Let us write X = I ] , where Xi and X4 are, respectively, 2x2 

\ Xs X4 J 

and {2g — 2) x (2g — 2) matrices. For each i > 2, we define Ai and Bi by 

A - ( \ r. _( h 

- V A )^^'-[o B, 

It is easy to see that, for i > 2, the equations XAi = AX and = -BjX 
imply that 

• X2Ai = X2Bi = X2, 

• AiX^ = BiXs = X3, and 

• X4Ai = AiXi and X^Bi = BiX^. 

It now follows from Lemma 12.31 and the assumption that all eigenvalues of 
X are equal to 1 that X2 = 0, X3 = and X4 = l2g-2- 

Moreover, from the equations XAi = AiX and XBiX = BiXBi, we 
obtain 

• XiU = UXi, and 

• XiUXi = UXiU. 

Now these two equations and Lemma 12.41 give us the equality Xi = U, so 
that X = Ai, which is the desired result. □ 

Remark 2.6. The above proof may be modified easily to prove the following 
version of Lemma 12.51 Suppose that 

(i) all eigenvalues of X G GL(2g, C) are equal to 1, 

(ii) XAi = AiX for ah i = 1, 2, . . . , 5, 

(iii) XBj = BjX for all 1 < j < 5 with j ^ k, and 

(iv) XBkX = BkXBk. 
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Then X = A^. The lemma holds true when the roles of Ai and Bi are 
exchanged as well. 

We state the following facts which will be used throughout the paper. 

Lemma 2.7. Let r and s be two positive integers. Then the subgroup of 



Lemma 2.8. The subgroup of GL(m, C) consisting of upper triangular ma- 
trices is solvable. 



Let S* be a compact connected oriented surface of genus g with a finite 
number of marked points in the interior. In this section we state the results 
from the theory of mapping class groups that are needed in the proofs of our 
main results. For further information on mapping class groups, the reader 
is referred to [10], or [1]. For a simple closed curve a on 5 we denote by ta 
the (isotopy class of the) right Dehn twist about a. 

Theorem 3.1. (|15j. Theorem 1.2) Let g > 1 and let a and b be two non- 
separating simple closed curves on S. Then there is a sequence 

a = ao,ai,02, ...,ak = b 

of nonseparating simple closed curves such that a^^i intersects Oj at only 
one point 

It is known that for g > 2, the group Mod(5) is generated Dehn twists 
about nonseparating simple closed curves. If the surface is closed this is due 
to Dehn [6j and Lickorish [18]. We record this fact and three well-known 
relations among Dehn twists. 

Theorem 3.2. If g > 2 then the mapping class group Mod(5) is generated 
Dehn twists about nonseparating simple closed curves on S. 

We note that the above theorem does not hold true for g = 1. More 
precisely, if S* is a torus with at least two boundary components, then Dehn 
twists about nonseparating simple closed curves are not sufficient to gen- 
erate Mod(S'): one also needs Dehn twists about the curves parallel to the 
boundary components [9l [T3] . 

Lemma 3.3. Let a and b be two simple closed curves on S, and let ta and 
tf) denote the right Dehn twists about them. 

(1) If a and b are disjoint, then ta and tb commute. 

(2) (Braid relation) If a intersects b transversely at one point, then 
they satisfy the braid relation tatbta = htatb- 



GL(r + s, C) consisting of the matrices of the from 




3. The mapping class group results 
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(3) (Lantern relation) Consider a sphere X with four boundary com- 
ponents a, b, c, d. Let x, y, z be three simple closed curves on X as 
shown in Figure Ul Then the Dehn twists about them satisfy the 
lantern relation 

ta^bic^d — txtytz- 




Figure 1 . The curves of the lantern relation. 

We remind that the first homology group Hi{G; Z) of a group G is isomor- 
phic to the abelianization G/[G,G], where [G, G] is the (normal) subgroup 
of G generated by all commutators xyx~^y~^ , where x,y £ G. 

Theorem 3.4. ([l^, Theorem 5.1) If g > 2 then first homology group 
Hi{Mod{S)]Z) is 

(1) trivial if g > 3, and 

(2) isomorphic to the cyclic group of order 10 if g = 2. 

If 5 ^ 3 then the group Mod(«S') is perfect. In the case g = 2, Mod(S') is 
not perfect but its commutator subgroup is perfect. 

Theorem 3.5. ([15], Theorem 4.2) If g > 2 then the commutator subgroup 
o/Mod(5) is perfect. 

Theorem 3.6. (|15j. Theorem 2.7) Let g > 2 and let a and b be two non- 
separating simple closed curves on S intersecting at one point. Then the 
commutator subgroup ofMod{S) is generated normally by taf^^ . 

Note that since any two Dehn twists about nonseparating simple closed 
curves are conjugate in Mod(S'), they represent the same element in the 
group i/i(Mod(S'); Z). In particular, we conclude the next lemma. 

Lemma 3.7. Let g > 1, and let b and c be two nonseparating simple closed 
curves on S. If H is an abelian group and if (j) : Mod(S') H is a homo- 
morphism, then 4>(th) = (j){tc). 

Let us write S = Sg^r, for the moment, for the surface of genus g >2 with 
p > boundary and with r > marked points in the interior. For r > 1 by 
forgetting one of the marked points, we get a short exact sequence, 

(1) 1 ^ 7ri(S^_,_,) Mod{Sl,.) Mod(S^_,_,) 1, 
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where the map from the fundamental group is obtained by pushing the base 
point along the given path. 

For p > 1, by blowing down a given boundary component d to a marked 
point, we get a short exact sequence 

(2) 1 ^ Z ^ Mod{Sl,) Mod(5^;ii) 1, 

where the group Z is generated by the Dehn twist about the bound- 
ary component d. These two exact sequences are called Birman's exact 
sequences. 





Figure 2. Various curves on the surface and a lantern. 



Let g >2, and let S = Sg Q be the closed surface obtained from S by gluing 
a disk along each boundary component. Let vr : Mod(5) — )• Mod(5') be 
the natural surjective homomorphism obtained by extending diffeomorphism 
5 — )■ S" to 5 — )• 5 by the identity on the disks glued. Let di,d2, ■ ■ ■ ,dphe (the 
simple closed curves parallel to) the boundary components of S. Consider 
the simple closed curves cq, ei, . . . , Cp, /o, fi, . . . , fr and a^_^ illustrated in 
Figure El Hence, the union di U Cj-i U ej bounds a pair of pants, fj-i U fj 
bounds an annulus with a marked point, and ag^iUa'g_-^ bounds a subsurface 
diffeomorphic to a torus with two boundary components. Clearly, for i = 
1,2, ... ,p and j = 1,2, ... ,r, the mapping classes tdi,tei_it'^^ and tf.^^fj^ 
are contained in the kernel of vr. It can be shown by using Birman's exact 
sequences that the kernel of vr is generated normally by 
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We state this fact as a proposition which will be useful for us. We also show 
that we may omit the Dehn twists about the boundary parallel curves. 

Proposition 3.8. The kernel of the natural surjective homomorphism ir : 
Mod(S') —7- Mod(5) is generated normally by the set 

(3) {te..,t-\tf^_,tl^ :l<i<p,l<j <r}. 

Proof. For the proof, it suffices to prove that each t^. can be written as a 
product of conjugates of elements in ([3]). We use the lantern relation for 
this. Consider the curves in the bottom picture of Figure [2l The curves 
di,ei,a,b bound a sphere with four boundary components. The curves on 
this sphere satisfy the lantern relation 

which may be rewritten as 

tdi = {te^_^t~^){txt~^){tyt'^'^). 

Since the pair (x,a) can be mapped (ej_i,ei) by a diffeomorphism of S, 
it follows that the element txt~^^ is conjugate to t(.i_^t~\ By the similar 
reasoning, tyf^^ is also conjugate to tei_^t~\ □ 

4. Eigenvalues and eigenspaces of 4){ta) 

Let (j) : Mod(S') — )• GL(m, C) be a homomorphism. For a simple closed 
curve a on S, we denote by La the image (j){ta) of the Dehn twist ta- If A is 
an eigenvalue of La, the eigenspace corresponding to A is denoted by E"^. 

For each i = 1,2, . . . , g, let ai and hi be the nonseparating simple closed 
curves on S shown is Figure [2j We may consider them on S as well, where S 
is the closed surface obtained from S by gluing a disk along each boundary 
component. The homology classes of the orient curves at and bi form a 
basis for the first homology group Hi{S,'L) of S. In order to avoid double 
subscript, we write L2i-i = (j){tai) and L2i = 4>{hi)- 

For an eigenvalue A of a matrix M, let A#(M) denote the multiplicity of 
A in the characteristic polynomial of M. We will omit M from the notation 
and write A# only; the matrix M will always be clear from the context. 

Lemma 4.1. Let L and M be two linear automorphisms ofC^ with LM = 
ML. If X is an eigenvalue of L, then ker(L — XI)^ is M -invariant for all 
k > 1. In particular, Ex = ker(L — XI) is M -invariant. 

Lemma 4.2. ([E]) Let (p : Mod(5) — )• GL(m, C) be a homomorphism. Let 
a, b, c, d be nonseparating simple closed curves on S such that there exists 
an orientation-preserving diffeomorphism / : 5 — )• 5 with f(c) = a and 
f{d) = b. Suppose that X is an eigenvalue of La = 4>{ta)- Then E'^ = E\ if 
and only if E^ = Ef. 
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Lemma 4.3. Let S he a compact connected oriented surface of genus g >1 
and let (j) : Mod(S') — )• GL(m, C) be a homomorphism. Suppose that there 
are two nonseparating simple closed curves a and b on S intersecting at one 
point such that E'^ = E\ for some eigenvalue A of La and L^. Then -E" is 
Mod{S) -invariant. That is, </)(/) (^a) = for all f £ Mod(5). 

Proof. Let x be a nonseparating simple closed curve on S. By Theorem 13. H 
there is a sequence a = oq, ai, 02, . . . , a/; = x of nonseparating simple closed 
curves such that aj_i intersects Oj at one point for all 1 < i < k. Since, by 
the classification of surfaces, there exists a diffeomorphism /j of 5 mapping 
(a, 6) to (ai_i,aj), we have E^"^'^ = E'^ by Lemma [4.21 It follows that 
Ef^ = E^ for all nonseparating simple closed curves x. Since Mod(S) is 
generated by nonseparating Dehn twists, we conclude that the subspace E^ 
is Mod(S')-invariant. □ 



Lemma 4.4. Let g > 3, let (j) : Mod(S') — )■ GL(m, C) he a homomorphism 
and let a be a nonseparating simple closed curve on S. Suppose that La has 
only one eigenvalue A with dim(£^") = m — 1. Then A = 1. 

Proof. Choose six nonseparating simple closed curves ci, C2, C3, C4, C5, cg on 
S disjoint from a such that we have the following lantern relation: 

Let a = {vi,V2, ■ ■ ■ , Vm} be a basis of C™ with Vj G E^ for j > 2, so that 



La 



A 



with respect to a. Since each Dehn twist tc^ is conjugate to ta, L^^ = 4>{tci) is 
conjugate to La and, hence, it has unique eigenvalue A. Since Lc^ commutes 
with La, it preserves the eigenspace E'^\ Lc^{E'^) = E^. Thus, the matrix of 
Lc^ with respect to the basis a is 

A 

* a 

where Ci is a square matrix of size m — 1. Now the lantern relation 

LaLciLc2Lc^ = Lc^Lc^Lcg 
implies that A^ = A'^, giving A = 1. □ 



Lemma 4.5. Let g > 3, let 4> : Mod(S') — t- GL(m, C) he a homomorphism 
and let a he a nonseparating simple closed curve on S. Lf ji is an eigenvalue 
of La = 4>{ta) with fi^ < 2g — 3 then jj, = 1 and the dimension of the 
eigenspace E^ is 
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Proof. If m < 2g — 1 then the image of (/) is trivial by Theorem [3l In 
particular, all eigenvalues of La are equal to 1. Now assume that m > 2g. 
Let /i = /io, /ii, /U2, . . . , /Us be all distinct eigenvalues of La- Set 



so that dim(i^) = /i^, and that C" = K K' . Let a be a basis K and a' 
be a basis of K' such that with respect to the basis a U a' of C"^, the matrix 
of La 



is in Jordan form, where ^ is a square matrix of size 

Let R be the complement of a regular neighborhood of a. Extending 
self-diffeomorphisms of i? to 5 by the identity gives a homomorphism q : 
Mod(ii) — )■ Mod(5'). Let tp = (pq and let a' be a simple closed curve on R 
which is isotopic to a on S. If / G Mod(i?) is any element, tp{f) commutes 
with La so that it preserves the subspaces K and K'. Hence, the matrix of 



where F is a square matrix of size 

Now, the correspondence f ^ F defines a homomorphism ip : Mod(i?) — t- 
Gh{K) = GL(/U#, C). Since ^# < 2g- 3 = 2(5 - 1) - 1 and the genus of R 
is 5 — 1 > 2, the image of the map tp is cyclic by Theorem [3l It is easy to 
find six simple closed curves 6, c, d, x, y, z which are nonseparating on R such 
that there is the lantern relation ta'thtcta = t^tytz, and that each of txf^^, 
tyt~^ and tzf^^ is a commutator in Mod(i?). Since ta' = ixfi^^tyf^^tzf^^ , 
it follows that ta' is contained in the commutator subgroup of Mod(i?). In 
particular, il>{ta') = I- As a result of this we have 



s 



K = ker(L, - , and K' = 0ker(L, - 



m 




Hf) is 





Hence, = 1 and dim(£^^) 



□ 



Corollary 4.6. Let g, (j) and a be as in Lemma \4.5\ If m < Ag — 5 then L, 
has at most two eigenvalues. 



a 
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4.1. The main lemma. The main step in the proof of Theorem [T] is the 
next lemma, which is also used in the proof of Theorem [2j 

Lemma 4.7. Let g > 1, m > 2g and let (p '■ Mod(S') — > GL(m,C) be a 
homomorphism. Let a be a nonseparating simple closed curve on S. Suppose 
that the Jordan form of La is 

1 1 
1 



(4) 











m-2 



Suppose also that there exists a nonseparating simple closed curve b inter- 
secting a at one point such that Ef ^ E\. Then there is a basis of C"^ with 
respect to which 



L„ 



A, 




and L, 







where J is the identity matrix of size m — 2g. (See Definition \2.1\ for Ai and 
Bi, and Figure\^for the curves Oj and bi.) 



Proof. For i = 1,2, 
L2i-l = 

We also set 



. . ,g, we set 

Lai, -^2i = Li)-, 



■2i-l 



Ai 



Ai 




and Bi 



E^' and E 



2i 



e\\ 



Bi 




Since E'^ ^ E\ and since there exists a homeomorphism mapping (a, b) to 
{ai,b.i), by Lemma we have S^i-i ^ all 1 < z < 5. Note that 

dim(£;-') = m - 1 for ah 1 < j < 2g. 

Since E^ / E'^, the dimension oiWi = E^CiE'^ is m—2. Let {vs, V4, . . . , Vm} 
be a basis of Wi. Choose vi e E^ \ E"^ and V2 G E'^ \ E^, so that 



Po = {vi,V2, ■ ■ ■ ,Vm} is an ordered basis of 
have 



With respect to /3o, we 





Xi 








1 





X 


Lm~2 



L, 



y'2 




1 



Y' 



Lrr 



where X = ( 2:3 x^ • • • Xm ) • 

If xi = then we conclude from the braid relation 

(5) L1L2L1 = L2L1L2 

that y'2 = 0. But then Li and L2 commute. Again from the relation ([5]) we 
get Li = L2, contradicting to E^ ^ E'^ . Hence, x\ is nonzero. 

Let w\ = x\V\ + X3f3 + xi^v^ + • • • + XmVm- Let /3q be the basis obtained 
from /3o by replacing vi with wi. With respect to this new basis we have 

1 



Li 



U 








m-2 



Au L2 



y2 



1 



Y 







Lrr 
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where Y = [ ■■■ ym Y ■ 

Prom the braid relation ([5|), it is easy to conclude that y2 = —1- Now let 
W2 = V2 — {ysv-s + 4 + • • • + ymVm) and let /3i be the basis /3q where V2 is re- 
placed with W2- With respect to the basis (3i = {wi, W2,V3, v^, . . . , Vn-i, Vm} 
we now have Li = Ai and L2 = B\. 

Suppose that k < g and that there is a basis 

Pk = {^1,^2, ■ ■ .,Vm-l,Vm} 

with respect to which 

(6) L2i-i = A and L2i = Bi 

for alH = 1, 2, . . . , A;. Note that in this case 

a = {v2k+l,V2k+2, ■ ■ ■,Vm-l,Vm} 
2k 

is contained in = . It can be shown easily that, in fact, a is a basis 

1=1 

for Wk, so that dim(Wfc) = m — 2k. 

Next, we consider L2k+i and L2k+2- Let s G {2k + 1, 2fc + 2}. Since the 
subspace is Lg-invariant, with respect to the basis j3k, 



Zs 



Here, Zg is a 2A: x 2k matrix. Since Lg commutes with each 

^2i-l = ( I j * ^ i / 

for i = 1,2, . . . ,k, where the matrix Ai is the 2k x 2k block diagonal matrix 
Diag(/2, . . . ,U, . . . , I2) whose i^^ block is U, and Bi is obtained from Ai by 
replacing U with U, we get that 

• ZgAi = AiZg, YgAi = Yg, 

• ZgBi = BiZg, YgBi = Yg 

for each i. We conclude from Lemma 12.31 that Zg = l2k and 1^ = 0, so that 



Lg 



hk 
Xg 



In particular, vi,V2,---, V2k are eigenvectors of Lg. 

If Wk were a subspace of E'^ , then we would have dim(i?'^) = m. By 
this contradiction, both of the subspaces Wk H E'^^^^ and Wk n £'2'=+2 of 
dimension m— 2A;— 1. If, furthermore, we had Wfcn£;2fc+i = Wfcn£;2fc+2^ then 
we would conclude that E'^'^+i = j^2k+2^ again arriving at a contradiction. 
Hence, the subspaces Wk H E'^fc+i and Wk H £;2fc+2 ^^^.^ different, so that 

Wk+i = Wk n E^^+^ n 

is of dimension m — 2k — 2. 
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Let {w2k+3,W2k+4:, ■ ■ ■ ,Wm} be a basis of Wk+i- We choose two vectors 
W2k+i and W2k+2 such that 

• W2k+1 G W^fc n y;2fc+l ^ W2k+1, 

• W2k+2 G Wfc n £2/0+2^ ^ W2k+1- 

Then {^2^+1, u^2fc+2, ■W2fc+3, ''^2fc+4, • • • , ^m-i, if^m} IS a basis of Wk- Now 
consider the basis 

A = {■^l,''^2, • • • ,'y2fc,'(i^2fc+l,'f^2fc+2, ■ ■ ■ ,Wm-l,Wm} 

of C". With respect to this basis, 

• L2i_i = Ai, L2i = Bi for i = 1,2,. .. ,k, 


• L2A;+i =1 Xi I and ^2^+2 



where X 





••• oy^_/io 

X2k+3 X2k+A ■■■ Xm 1 ' ^ \ U' 



I I / \ t 

y2k+3 y2k+4 ■ ■ ■ Vrri 



1 

^^^^2=, Q ••• 

The rest of the proof proceeds as above: If x = then the braid relation 

(7) L2fc+l-^^2fc+2-^^2fc+l = L2k+2L2k+lL2k+2 

would imply that y' = 0, so that L2k+i and L2k+2 commute. The braid 
relation ([7]) now gives ^2^+1 = -^^2fc+2) which is a contradiction. 
Hence, x 7^ 0. Define 

''^2fc+l = XW2k+l + {x2k+3W2k+3 + X2k+4W2k+i H h XmWm) 

and let /3[„ be the basis obtained from by replacing w;2fc+i with w'2j._^i- 
With respect to the matrices of Li, . . . , L2k are the same, and ^2^+1 = 
Akj^i. The matrix of L2k+2 turns into a new matrix of the form above 
where y' is replaced by some y, and y'j is replaced by some yj. The braid 
relation d?]) then implies that y = —1. If we now define 

W2k+2 = W2k+2 - iy2k+3W2k+3 + y2fc+4l«2fc+4 H \- V 

and 

Pk+l ={vi,V2,..., V2k,W2k+l,'w'2k+2^W2k+3, W2k-i, Wm-l,Wm}, 

we have -^24-1 = Ai and L2i = Bi for all 1 < i < A; + 1 with respect to /Sfc+i- 
Consequently, repeating this for k = 1,2,3, . . . , g — l, with respect to some 

basis of C™, ^21-1 = Ai and L2i = Bi for al\ 1 < i < g. 

This finishes the proof of the lemma. □ 
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4.2. Triviality of a representation of mapping class group. We give 
some criteria for the triviality of a representation of the mapping class group 
into GL(m, C). The main tool for this is Theorem [3l This subsection is 
inspired by [7]. 

Lemma 4.8. Let S he a compact connected oriented surface of genus h > 2 
and let tp : Mod(S') — GL(?ti, C) be a homomorphism. If there is a flag 
= Wo C Wi C W2 C • • • C Wk = C™ of Mod{S) -invariant subspaces such 
that dim(Wj/Wj_i) < 2h — 1 for each i = 1,2, . . . ,k, then the image of is 

(i) trivial if h > 3, and 

(ii) a quotient o/Zio if h = 2. 

Equivalently, the image of the commutator subgroup of Mod(S') is trivial. 

Proof. We set F = Mod(S'). For each i = 1, 2, . . . , /c, let be a basis of Wi 
such that ai C ai+i- We work with the basis of Wk = C™. Since each 
Wi is F-invariant, for f £T the matrix </>(/) is of the form 














F2 












F3 ■ 




V 





• 





Then, for each i, the correspondence f ^ Fi defines a homomorphism ipi : 
F Gh{Wi/Wi-i). Since dim(Wi/VFi_i) <2h-l, the image oiipi is cyclic 
(trivial if /i > 3) by Theorem [3l 

Since the image ijji is abelian, we get ipi{f) = / for all / G [F,F] and for 
all i. Hence, for any / G [F,F], ip{f) is upper triangular with 1 along the 
diagonal. The subgroup of GL(m, C) consisting of such matrices is nilpotent, 
and the group [F,F] is perfect by Theorem 13.51 We conclude from this that 
ip{\r,T]) is trivial. □ 

Corollary 4.9. Let R he a compact connected oriented surface of genus 
g — 1 > 3 and let tp : Mod(i?) — t- GL{2g, C) be a homomorphism. If there 
is a Mod{R) -invariant suhspace W with 3 < dim(M^) < 2g — 3, then ij) is 
trivial. 



5. Uniqueness of the symplectic representation 

Let 5 be a compact oriented surface of genus g > 3 and let 4> : Mod(S') — )■ 
Gh{2g, C) be a homomorphism. We prove in this section that either (p is 
trivial or, with respect to a suitable basis of C^^, the image of (p is equal to 
Sp(2g', Z) with respect to a suitable basis of C^^. Theorem [T] will follow from 
this. Recall that for a simple closed curve a, denotes the eigenspace cor- 
responding to an eigenvalue A of = (p>{ta) and A# denotes the multiplicity 
of A. 



SYMPLECTIC REPRESENTATION OF MAPPING CLASS GROUP 



15 



w 



Lemma 5.1. Let 5 > 4, let a he a nonseparating simple closed curve on S 
and let A be an eigenvalue of La- If > 3 then dim(£'^) > 2g — 1. In 
particular, A# > 2g — 1. 

Proof. Let 6 be a nonseparating simple closed curve on 5 intersecting a at 
one point, and let R denote the complement of a regular neighborhood of 
aUb, so that it is a subsurface of genus g — 1- Then, Mod(i?) injects into 
Mod(S'). By identifying Mod(i?) with its image, we assume that Mod(i2) is 
a subgroup of Mod(5). 

Suppose first that dim(£'^) < 2g — 3. Define a subspace W by 

kei{La - M)^* , ifX#<2g-3, 

ker(La - Xlf, if A# > 2^ - 2 and dim(^^) = 1, 

ker(La - Xlf, if A# > 2^ - 2 and dim(£;^) = 2, 

E'^, if A# > 25 - 2 and 3 < dim(^^) < 25 - 3. 

Since elements of Mod(i?) commute with the Dehn twist ta, the subspace W 
is Mod(i2)-invariant and its dimension satisfies 3 < dim(VF) < 2{g — 1) — 1. 
Since the genus of is g — 1 > 3, (p{Mod{R)) is trivial by Corollarv 14.91 
Since ta is conjugate to some Dehn twist in Mod(i?), we get that La = I. 
This says, in particular, that dim(£'") = 2g, which is a contradiction. 

Suppose now that dim(^^) = 2g - 2. If / E\, then E^ C] E\ \s a. 
Mod(i?)-invariant subspace and its dimension is either 2g — 3 or 2^' — 4. 
Hence, (p is trivial on Mod(i?) by Corollary 14.91 We conclude again that 
La = L, obtaining a contradiction. If E'^ = E^ then by Lemma 14.31 the 
eigenspace E'^ is a Mod(S')-invariant subspace of dimension 2g — 2. Hence, 
C -B^ C C^^ is a Mod(S')-invariant flag. Now Lemma 14.81 applies to 
conclude that (j) is trivial, arriving at a contradiction again. □ 

Lemma 5.2. Let g > 4, let a be a nonseparating simple closed curve on S 
and let A be an eigenvalue of La- /f dim(£'") = 2^ — 1 then A = 1. 

Proof. Let a = ci. Choose six nonseparating simple closed curves 02,03,04,05, oe, 07 
on S such that we have a lantern relation 

(8) ici^C2^C3^C4 — icsicfjicT 

Hence, 

(9) Lc^Lc2LczLc^ = Lc^LcqLc-j. 

The subspace nl=i -^a positive dimension. Let be a nonzero vector 
in this intersection. Evaluating both sides of ([9]) to u gives X'^v = X^v, 
concluding A = 1. □ 

5.1. Proof of Theorem [1] for 5 > 4. We give the proof in several steps. 
Let a be a nonseparating simple closed curve on 5 and let La = (p{ta)- 

Step 1: We claim that La has only one eigenvalue. First of all. La has at 
most two eigenvalues by Corollary 14.61 Suppose that it has two eigenvalues 
A and with A# > Since A# + = 2g, we have < g and A# > g. 
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Then, we get from Lemma 14.51 that = 1, so that A 7^ 1. On the other 
hand, by Lemma [5 -H A# = 2g—l and the dimension of the eigenspace is 
2g— 1. Now Lemma [5.21 imphes that A = 1, giving the desired contradiction. 
Therefore, La has only one eigenvalue, say A. 

By Lemma [5 .11 the dimension of the eigenspace E'^ is either 2(7 — 1 or 2g. 

Step 2: Suppose first that dim(£'^) = 2g, i.e. La = XL For any non- 
separating simple closed curve x, the Dehn twist tx is conjugate to ta, so 
that Lx is conjugate to La, implying that L^ = XL Since the mapping 
class group Mod(5) is generated by Dehn twists about nonseparating sim- 
ple closed curves, it follows that the image of (j) is cyclic. Since the mapping 
class group Mod(S') is perfect, the image of (p is trivial. 

Step 3: Suppose finally that dim(£'^) = 2g — 1. Lemma 15.21 implies 
that A = 1. Thus, the Jordan form of La is as in (j4|). Now by Lemma |4.7^ 
with respect to a basis of C^^, we have La. = Ai and L^- = Bi for each 
i = l,2,...,g. 

Step 4: Let S be the surface obtained from S by gluing a disk along 
each boundary component and let vr : Mod(S') — t- Mod(5) be the surjec- 
tive homomorphism obtained by extending diffeomorphisms by the iden- 
tity over the glued disks. Consider the curves illustrated in Figure [2j Let 
e € {eo, ei, . . . , Cp, /o, /i, . . . , fr}- Since e intersects bi at one point and is 
disjoint from all a, and bj, j > 2, the matrix Le = (p{te) commutes with 
all Ai and all Bj, and satisfy the braid relation LgBiLf, = BiL^Bi. Since 
Lg is conjugate to La, it has only one eigenvalue A = 1. From Lemma |2.5^ 
we get that Lg = Ai. It now follows from Proposition 13.81 that the kernel 
of vr is contained in the kernel of (p. Hence, (p induces a homomorphism 
^ : Mod(5) ^ GL{2g,C). 

Step 5: Note also that J = Ag^i = 4>{tag-i), giving vr(t;^^i_^^_,) = 

/. A celebrated result of Johnson [11] says that the Torelli subgroup of 
Mod(S) is generated normally by 7r(t~^^_^ta/ ^). Therefore, the Torelli sub- 
group of Mod(5^) is contained in the kernel of (p, so that (p induces a map 
^:Sp(2g,Z)^GL(2<7,C). 

Mod(5) 
Mod(5) 
Sp(29,Z) 

Step 6: As the matrix La has infinite order, lm{ip) = lm{(p) is infinite. 
One may easily check that {A1B1A2B2 ■ ■ ■ AgBg)^ = —I so that the kernel 
of (p does not contain — /. From the solution of the congruence subgroup 
problem for Sp(2(7, Z) (see [H], Corollar 1.), we conclude that the kernel of 
ip is trivial, so that ip is injective. 

This concludes the proof of Theorem [H 
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5.2. Sketch of the proof of Theorem [T] for g = 3. The proof of The- 
orem [T] for g = 3 requires more detailed analysis of eigenvalues. We only 
sketch the proof. Note that in this case cp '■ Mod(5) — )■ GL(6,C). 

Let a and b be two nonseparating simple closed curves intersecting at one 
point, let R be the complement of a regular neighborhood of a U 6, and let 
ci and C2 be two simple closed curves on R intersecting each other at one 
point. 




Figure 3. Curves on the surface of genus 3. 

In this case, La has at most two eigenvalues by Corollary 14.61 again. Sup- 
pose first that La has two eigenvalues A and fi with A# > Then A# > 4, 
/x^ < 2, fi = 1 and dim(£'^) = /i^. Let r = dim(£^"). Consider the following 
Mod(i?)-invariant flags: 

• C ker(La - A/)^ c if r = 1, 

• C ker(La - XI) C ker(La - A/)^ c if r = 2, 

• C C if r = 3. 

Therefore, in the cases r < 3, Lemma 14.81 gives that the commutator sub- 
group of Mod(i?) is in the kernel of (p. In particular, (j){tc^t~^) = L Since 
the normal closure of tcit~^ in Mod(5) is Mod(5) (c.f. Theorem 13. 6p . the 
map (j) is trivial, so that La = I, a. contradiction. In the case r = 4, the 
flag C n C C is Mod(ii)-invariant if ^ E\, and the 
flag C -E"^ C is Mod(S')-invariant if E"^ = E"^, both implying that (p is 
trivial, a contradiction again. 

Suppose now that r = 5. Fix a basis of such that La = 

Since the first homology group of Mod(S) is trivial, A^ = 1. For any simple 

closed curve x disjoint from a, the matrix of Lx is equal to L^ = 

for some nonzero complex number 7(3;). If x is a nonseparating curve, then 
7(x) is either 1 or A, as it is an eigenvalue of Lx- 

Consider the curves illustrated in Figure [3l If 7(01) = 1 then E'l = Ef. 
By the braid relation, it can be seen that 7(02) = 1. Hence, E^^ = Ef. It 
follows that Ef = Ef for all nonseparating x on S, so that Ef is Mod(S')- 
invariant. One may deduce from this that (j) is trivial. In particular, A = 1. 
If 7(ci) = A then braid relations between imply that 7(02) = 7(03) = A. 
Now the relation (tciic2*c3)^ = iata' and A^ = 1 imply that 7(0') = A^, which 




( 7(^) ) 
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is an eigenvalue of La' ■ On the other hand, since all eigenvalues of La' are 1 
and A, we conclude that A = 1. By these contradictions, La must have only 
one eigenvalue, say A. 

If dim(£'^) < 4 then one arrives at a contradiction by deducing that </> 
must be trivial. If dim(£'") = 6 then (p is trivial. If dim(£'") = 5 then 
Lemma 14.41 gives A = 1. Now the rest of the proof proceeds as in the case 
g>4. □ 



6. Non-faithful representations of Mod(5): Proof of 

Theorem [2] 

Our aim in this section is to prove Theorem [2j We do this by proving a 
stronger result, Theorem 16. 1|, which says that certain terms in the derived 
series, defined below, of the Torelli subgroup of the mapping class group of a 
subsurface of genus three are contained in the kernel of (p. Since the Torelli 
subgroups of mapping class groups we consider are not solvable (they contain 
nonabelian free groups). Theorem 16 . 1 1 will certainly imply Theorem[2j Recall 
that the Torelli subgroup of the mapping class group of an orientable surface 
with at most one boundary component is the subgroup consisting of those 
mapping classes which act trivially on the first homology of the surface. 

For a subsurface X oi S with one boundary component, the inclusion 
X ^ S induces a homomorphism M.od{X) — )■ Mod(S'). This map is injective 
when the genus of X is less than that of S. When this is the case, we identify 
the group Mod(X) with its image in Mod(S') as usual. We denote by Tx 
(the image of) the Torelli subgroup of Mod(X). 

For a group G, let G^^"^ = G. For each integer k > 1, we inductively define 
the k^^ derived subgroup G^'^) of G by 

Q{k) ^ [G'{'=-i),G'(fc-i)]. 
Recall that a group G is called solvable if G^''^ = 1 for some k. 

Theorem 6.1. Let n > be an integer. If g > n + 2> and if (p : Mod(S') — t- 
Gh(2g + n, C) is a homomorphism, then the n^^ derived subgroup of 
Tx is contained in the kernel of (p for any genus-3 subsurface X of S with 
one boundary component. 

Proof. We prove the theorem by induction on n. If n = then g > 3, so the 
result follows from Theorem [TJ 

Let n > 1, 5 > n + 3 and let (p : Mod(S') GL(25' + n, C) be a homo- 
morphism. Suppose that the theorem holds true for all /c < n — 1. If X and 
Y are any genus~3 subsurfaces S each with one boundary component, then 
the classification of surfaces tells us that there is a self-diffeomorphism of 
S mapping X to Y. It follows that the subgroups Mod{X) and Mod(y) of 
Mod(5) are conjugate. It is now easy to conclude that it suffices to prove 
the theorem for some genus~3 subsurface. 
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Let a be a nonseparating simple closed curve on S. Since 2g + n < ig — 5, 
by Corollary 14.61 the matrix La has at most two eigenvalues. 

Let 6 be a nonseparating simple closed curve intersecting a at one point 
and let R denote the complement of a regular neighborhood of aU 6, so that 
i? is a compact surface of genus g — 1- By extending self-diffeomorphisms R 
to S by the identity, we regard Mod(i?) as a subgroup of Mod(5). 

Case 1: La has two eigenvalues. Suppose that La has two eigenvalues 
A and /x, with A# > Recall that A# denotes the multiplicity of the 

eigenvalue A. Since A# + = 2^ + n < 3(7 — 3, we have < 2g — 3. 
By Lemma 14.51 we have /i = 1 and dim(£'^) = fi^. The dimension of 
ker(La — XI)^# is A#. Let ai be a (ordered) basis of ker(La — XI)^# and let 
02 be a basis of E^. Then a = ai U a2 is a basis of C^^"*"" and in this basis 
the matrix of La is of the form 

(10) 

where the size of the identity matrix / is We work with this basis. 

For any / G Mod(i?), (/)(/) commutes with La so that the subspaces 
ker(La — XI)^* and = ker(La — /) are (/)(/)-invariant, so 

where F' is of size This way we get a homomorphism ip : Mod(i?) — t- 
GL(/x#,C) defined by (p{f) = F'. Note that since n > 1, the genus of S 
satisfies g > 4, so that the genus of i? is (7 — 1 > 3. By applying Theorem [3] 
to ip, we conclude that F' = /. It then follows that for any simple closed 
curve X that is nonseparating on R, the eigenspace of L^ satisfies Ef = Ef. 
In particular, L^ is of the from (jlOp . 

If c and d are two nonseparating simple closed curves on R intersecting 
at one point, then we have Ef = Ef (= Ef). Now apply Lemma 14.31 to 
conclude that Ef is Mod(5')-invariant. Since the dimension of E^ is less 
than 2g, the action of Mod(5) on Ef is trivial. In particular, for any element 
/ E Mod(5), the matrix of 4>{f) is 

It follows that (j) gives rise to a homomorphism (f) : Mod(5) — )• GL(A#,C) 
defined by 

*(/) - ( ''p ; ) . 

Since A# < 2^ + n — 1, by the induction hypothesis is contained in 

ker((^) for some genus-3 subsurface X of S. We conclude from this that 
4>{Tx is abelian, and hence, (p{T^'^) is trivial. 
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Case 2: La has only one eigenvalue. Suppose that La has only one eigen- 
value, say A. Let X be any genus-3 subsurface of R with one boundary 
component. We may consider Mod(X) as a subgroup of Mod(i?). 

We claim that if there is a Mod(i?)-invariant subspace V of dimension 
ri with 3 < 7'i < 251 + n — 3, then we are done, namely 4){T^^) is trivial. 
For the proof of the claim, suppose that we have such a subspace V . Let 
r2 = 2g + n — ri and let o; be a (ordered) basis of C^^^" such that first ri 
elements span V . With respect to this basis, for any / G Mod(i?), 



Mf) 



so that we have two homomorphisms : Mod{R) — >■ GL(y) = GL(ri,C) 
and (j)2 ■■ Mod(i?) ^ GL(C25+"/y) = GL(r2, C). Since both n and ra satisfy 
1^1 ^ 2((7 — 1) + n — 1 and r2 < 2(g — 1) + 77, — 1 and since g — 1 > {n — 1) + 3, 

the derived subgroup of Tx is contained in the kernel of both (pi by 

the induction hypothesis. That is to say, 

L 

for any / G T^~'^\ It follows that (j){T^~^^) is abehan, and hence (piT^) 
is trivial, proving the claim. 

Let us set r = dim(£'^) for the rest of the proof, so that 1 < r < 2g + n. 

Suppose first that r < 2g + n — 3. If r is equal to 1 or 2, then 3 < 
dim(ker(La — AI)^) < 6. It follows that in this case, there is a Mod(i2)~ 
invariant subspace V with 3 < dim(y) < 2^ + 77 — 3. Hence, by the above 
claim, (j){T^^) is trivial. 

Suppose next that r = 2g + n, so that La = XL Since tx is conjugate to ta 
for any nonseparating simple closed curve x on S, L^ = XL Since Mod(S') 
is generated by such Dehn twists, it follows that the image of (j) is cyclic. 
Since the first homology group of Mod(S') is trivial, we conclude that the 
image of (j) is trivial. 

Suppose now that r = 2g+n—2. The dimension of is also r. If E'^ 7^ 
then El n E\ is a Mod(-R)-invariant subspace of dimension 2g + n — 3 or 

(n) 

2(7 + 77 — 4. Hence, Tj^ is contained in the kernel of cp by the claim above. 
If El = El then it follows from Lemma |43] that El is Mod(5)-invariant. 
Let /3 be an ordered basis of C^^"*"" whose first r elements form a basis of 
El- For any / G Mod(S'), the matrix of (j){f) with respect to /3 is 



Hf) 



Mf) * 
Mf) 



In this way we get two homomorphisms (f>i : Mod(5) — )• GL(r, C) and (p2 ■ 
Mod(S') — )■ GL(2,C). By Theorem [3l the image of (1)2 is trivial, and by the 

induction hypothesis, is contained in the kernel of Therefore, for 
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It follows that (piTj^ ^'') is abelian, and 4'{T^ ^'^) is trivial. In particular, 
(piT^) is trivial. 

Suppose finally that r = 2g + n — l. By Lemma [4.41 we get A = 1. If Ef = 
E\ then by Lemma 14.21 we get that Ef = Ef for all nonseparating simple 
closed curves x on S. Since the group Mod(5') is generated by Dehn twists 
about nonseparating simple closed curves, it follows that every element of 
Mod(S') acts trivially on Ef. With respect to a basis of C^^^" whose first 
r = 2g + n — 1 elements belong to Ef , the matrices of La and Lj, are of the 
form 

Ir * 
1 

It follows that LaLb = LbLa- From the braid relation LaLbLa = LbLaLb, 
we get La = Lb, and so (j){tat'^^) = I- Since the normal closure of taf^^ 
in Mod(S') is the whole group, we conclude that (j) is trivial, which is a 
contradiction to dim(£'") = 2g -\- n — \. 

Therefore, we have Ef ^ E\. Now, by Lemma 14. 7| we have La^ = 

In ) ^'' = [ In 

now be concluded as in the proof of Theorem [1] that the kernel of Mod(S') — t- 
Sp(2g', Z) is contained in the kernel of (j). In particular, Tx, and hence 
is contained in the kernel of 4>. 

This completes the proof of the theorem. □ 



and Lf,- = \ J ^ I for a suitable basis of C^^"*"". It can 
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